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Executive Summary 


The Council of Independent Colleges (CIC) has long been concerned about the educational success of 
students, particularly underrepresented groups of students. As part of its Creating Pathways to Educational 
and Economic Opportunity in Urban Colleges and Universities project (the Pathways Project), CIC organized 
19 institutions, nine in urban and ten in non-urban areas, to explore students’ performance on the Collegiate 
Learning Assessment (CLA). In addition to collecting data from a representative sample of students, participating 
institutions agreed to draw an in-depth sample of first-generation students (defined based on parental education) 
and low-income students (defined based on Pell-grant eligibility). The in-depth sample increased the sample size 
of underrepresented groups and thus allowed for more accurate estimates of their CLA performance. The data 
included in this report are cross-sectional, with institutions collecting information from samples of first-year 
students in fall 2010 and seniors in spring 2011. 

The first portion of this report focuses on CLA performance of underrepresented groups of students. Though 
descriptive results reveal gaps in CLA performance between underrepresented groups and their more advantaged 
peers, those differences can be accounted for by student characteristics. Thus, after adjusting for student 
characteristics, particularly academic preparation, no notable gaps in CLA performance appear among different 
groups of students. More specifically: 

■ Descriptive results indicate that first-generation students perform less well on the CLA than non-first- 
generation students. These two groups of students differ along multiple dimensions, however. After 
students’ individual-level characteristics are considered, particularly their academic preparation, there are 
no differences in CLA performance in either first or senior year between first-generation and non-first- 
generation students. 

■ Similarly, after adjusting for individual-level characteristics, there are no differences in CLA performance 
in either first or senior year between Pell-eligible and non-Pell-eligible students. 

■ Descriptive gaps in CLA performance by race/ethnicity are substantially larger than those by first- 
generation status and Pell-grant eligibility. African-American and Hispanic students score substantially 
lower on the CLA at entry into college than their white peers. Individual-level characteristics, and 
particularly academic preparation, however, explain gaps between African-American/Hispanic and 
white students. After adjusting for individual-level differences, there are no racial/ethnic gaps in CLA 
performance in either first or senior year. 
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■ The same patterns are observed in both urban and non-urban settings: After adjusting for students’ 
individual-level characteristics, there are no differences in CLA performance between examined groups of 
students in either urban or non-urban settings. 

The second portion of the report focuses explicitly on CLA performance of both students and institutions in 
urban and non-urban settings. These analyses reveal several key findings: 

■ Student-level analyses reveal no difference in CLA performance in urban and non-urban settings, after 
adjusting for students’ individual-level characteristics. One exception is the performance task of the CLA 
in the senior year, when students in urban contexts seem to perform less well than students in non-urban 
settings. 

■ Institutional value-added analyses confirm results obtained from individual-level models. On average, 
urban institutions have slightly lower value-added scores than non-urban institutions with respect to the 
performance task measure, but not with respect to other components of the CLA. 

■ Institutional value-added analyses also reveal large variation among institutions within both urban and 
non-urban contexts. There are institutions in each context that have positive and negative value-added 
scores. This variation within each setting by far overshadows any overall differences between urban and 
non-urban settings. 

These findings have several notable implications: 

■ The most important predictor of CLA performance is academic preparation. After controlling for 
individual-level characteristics, particularly academic preparation, there are no differences in CLA 
performance among different groups of students. This finding holds in both urban and non-urban 
institutional settings. Therefore, this finding suggests that postsecondary institutions should work with 
local high schools to improve student preparation or influence broader state and national conversations to 
prepare high school students for college-level work. 

■ Urban and non-urban institutions on average perform equally well on the CLA, and most of the 
variation is within specific institutional settings (i.e., urban and non-urban) as opposed to across 
them. Urban and non-urban institutions do equally well in educating the students they enroll (as assessed 
by the CLA). They also do equally well in educating different groups of students, including students from 
underrepresented groups. Each setting, however, includes institutions with higher and lower value-added 
scores. Individual institutions, not broad categories such as urban and non-urban, thus present a more 
productive focus of analysis and policy development. 
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■ CLA performance varies more within institutions than across them. All institutions have students 
who perform at different levels, producing much more variation within institutions than across 
them. Institutions would thus benefit from focusing within — studying carefully their own students and 
understanding what groups of students perform well and how their successes can be replicated, as well as 
what groups of students do not perform as well and the specific strategies that are needed to improve their 
outcomes. 

Finally, this report could not have been written without the concerted effort of many institutions. Moreover, 
institutions cannot improve their outcomes without understanding the challenges and successes of their students. 
To better examine variation on student learning outcomes within an institution, additional assessment measures 
beyond those examined in this study will be needed. Investing in institutional infrastructure to collect high- 
quality assessment data is thus an important part of the puzzle of improving learning outcomes for all students. 
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I. Introduction 


Background and Context 

The Council of Independent Colleges (CIC), an association of more than 600 small and mid-sized independent 
colleges and universities, has long been a national leader in voluntary efforts to improve the quality of student 
learning and a strong advocate of institutional autonomy in accountability efforts. Since 2002, CIC has 
collaborated with the Council for Aid to Education (CAE) to develop and implement the Collegiate Learning 
Assessment (CLA), one of the first standardized instruments to measure directly an institution’s contribution to 
student learning. In 2002, CIC helped CAE identify smaller private colleges to test the prototype of the CLA. 

In 2003, CIC recruited 12 member colleges and universities to form the CIC/CLA Consortium and participate 
in the first year of public use of the CLA. The following year, CIC expanded this initial Consortium to include 
33 colleges and universities with a three-year commitment (2005-2008) to use the CLA. In its third phase, 47 
CIC institutions collaborated in another three-year commitment (2008-2011). For the most recent report about 
the CIC/CLA Consortium, see Catalyst for Change: The CIC/CLA Consortium (2011) at www.cic.edu/Catalyst-for- 
Change. 

In this study, CIC is concerned about the educational success of students who attend urban institutions. There is 
mounting evidence that it may be especially important to encourage urban independent colleges and universities 
to engage in serious assessment of student learning. A great many students, particularly low-income and minority 
students, are entering urban universities but never completing a degree. More than 40 percent of CIC member 
institutions are located in urban settings and serve predominantly urban students. In 2009, CIC was awarded 
a grant by the Carnegie Corporation of New York to build on the efforts of the CIC/CLA Consortium and 
gain insight into the factors that contribute to (or detract from) the academic achievement of students at urban 
institutions and share information on effective strategies. In this way, CIC intends to create new pathways to 
educational and economic opportunity for students who attend urban colleges and universities, many of whom are 
from historically underserved populations. Thus this program is officially titled Creating Pathways to Educational 
and Economic Opportunity in Urban Colleges and Universities (hereafter referred to as the Pathways Project). 

Ten additional CIC member colleges and universities joined 19 institutions that had previously participated in the 
CIC/CLA Consortium to form the Pathways Project (2009-2012). 

As part of the Pathways Project, CIC arranged for the 19 institutions — nine in urban settings and ten in non- 
urban locales — that had previous experience in the CIC/CLA Consortium to participate in a study of the learning 
outcomes of underrepresented students. “Urban” was defined as a city with a population greater than 100,000 or a 
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large suburb located in close proximity to a major metropolitan area where the institution serves a high percentage 
of students from that metropolitan area. The key outcome of interest was students’ performance on the CLA, a 
standardized assessment that aims to capture higher-order collegiate skills, including critical thinking, complex 
reasoning, and written communication. The CLA provides four different scores: performance task, make an 
argument, break an argument, and argument (which is the average of make an argument and break an argument 
scores). Throughout most of this report, results for all four CLA components are reported. In general, the patterns 
are consistent across measures, increasing confidence in the reported results. 

Participating Institutions and Sampling 


The 19 institutions that participated in this study agreed to administer the CLA to their first-year students and 
seniors in academic years 2010-2011 and 2011-2012. The assessment was given to approximately 100 first-year 
and 100 senior students, representative of the student population at each institution. The CLA was administered 
using a sampling method where roughly half of the students took the performance task components and the other 
half took the argument components of the CLA. To get reliable estimates of the performance of underrepresented 
students, institutions had to test additional students. Therefore, in 


addition to the representative student samples, these 19 institutions 
were also asked to draw an in-depth sample of first-generation and 
low-income first-year and senior students of up to 100 for each 
group to take the CLA in the academic year of 2010-2011. 

This report analyzes the results from the 2010-2011 CLA 
administration at the 19 participating institutions. The data 
included in this report are cross-sectional, with institutions 
collecting information from samples of first-year students 
in fall 2010 and seniors in spring 2011. The study includes a 
representative sample of 2, 645 first-year students and 1,999 
seniors and an in-depth sample of 834 first-year students and 441 
seniors who were first-generation and/or low-income students (see 
the Methodological Appendix for the number of students that 
participated at each institution). Most of the analyses presented 
are based on the full sample (i.e., the combined sample including 
students in both the representative sample and the in-depth 
sample). Differences between the two samples are discussed in the 
Methodological Appendix, as are definitions of all of the variables 
used in sampling and analysis. The final set of analyses examining 
institutional performance and value-added scores is based on the 
representative sample. 


PARTICIPATING INSTITUTIONS 

Institutions in Urban Areas 

Alaska Pacific University 
Bethel University (MN) 

Cabrini College (PA) 

Carlow University (PA) 

Charleston Southern University (SC) 
Dominican University (IL) 

Notre Dame of Maryland University 
Trinity Christian College (IL) 
University of Charleston (WV) 

Institutions in Non-Urban Areas 

Barton College (NC) 

College of Saint Benedict/ 

Saint lohn’s University (MN) 
Illinois College 
Indiana Wesleyan University 
Lynchburg College (VA) 

Morningside College (IA) 

Stonehill College (MA) 

Texas Lutheran University 
University of Great Falls (MT) 
Westminster College (M0) 
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II. CL A Performance of Students by First- Generation Status 


The report begins by examining variation in 
CLA performance between first-generation and 
non-first-generation students. “First-generation” 
is used to designate students whose parents did 
not complete college (for further information on 
this definition, please see the Methodological 
Appendix). Descriptive results in Figure la 
show that first-generation students enter higher 
education with lower levels of skills as measured 
by the CLA. More specifically, first-generation 
first-year students have lower performance on all 
components of the CLA compared to their non- 
first-generation peers. 1 All of the differences 
are statistically significant, but not of large 
magnitude (see Table 1A in the appendix for effect 
size estimates)? First-generation seniors still score lower on the CLA than their non-first-generation peers, but the 
gaps among seniors are approximately half the size of those among first-year students (see Figure lb). 

Based on these descriptive results, it may be tempting to conclude that the gaps in performance between first- 
generation and non-first-generation students decrease during their time in college. But that conclusion does 
not seem warranted. First-generation students tend to differ from non-first-generation students along multiple 
dimensions. Table 1 illustrates the difference in one important characteristic: pre-college academic performance. 
The results indicate that first-generation students have substantially lower pre-college academic performance than 
non-first-generation students (the gap is approximately half a standard deviation). 

Moreover, since this report relies on cross-sectional data, the same students are not observed twice, raising the 
possibility that the senior sample differs from the first-year student sample. Results in Table 1 show a slight 
increase in pre-college academic performance between the first-year and senior samples in both urban and non- 

^ The y-axis range in Figures la and lb (and all similar figures reporting average CLA scores) is approximately two standard deviations. 

2 

Typicaiiy, the effect size up to 0.3 standard deviations wouid be regarded as smaii, around 0.5 as medium, and 0.8 and above as iarge. Aii 
of the differences between first-generation and non-first-generation students are at or beiow 0.3 standard deviations. 


Figure la. First-Year CLA Scores, by First-Generation Status 
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urban settings. Although the difference is not 
large, which is reassuring for the comparisons 
of CLA performance across the two samples, 
it does not preclude the possibility that the two 
samples differ along other dimensions, requiring 
caution when interpreting results based on a 
cross-sectional study design. Furthermore, 
the limitations of this study do not permit 
determining if the difference in academic 
performance between first-year and senior 
students is attributable to differential student 
enrollment patterns between the two classes, 
attrition from the first to the senior year, or 
response bias among seniors that took the CLA. 


Figure lb. Senior CLA scores, by First-Generation Status 
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Given differences between first- 
generation and non-first-generation 
students in individual-level 
characteristics, it is important to 
adjust CLA estimates for those 
differences. Although no dataset 
includes variables describing all 
of the ways in which the two 
groups may differ, adjusting for 
observable characteristics available 
in the dataset is an important 
step in this process. This report 
includes regression analyses 
of different components of the 
CLA, controlling for students’ 
sociodemographic characteristics 
(gender, race/ethnicity, parental 
education, and an indicator of 
English as a primary language) and 
pre-college academic preparation 
(see Table 2A in the appendix for 
more details and alternative model 
specifications). 


Table 1 . Students’ Pre-College Academic Performance, 
by First-Generation Status and Urbanicity 



Non-First-Generation 

First-Generation 

Difference 

TOTAL 




First-year students 

1,090.24 

1,015.03 

75.21 


(153.37) 

(150.21) 


Seniors 

1,105.78 

1,037.51 

68.27 


(160.12) 

(152.49) 


NON-URBAN 




First-year students 

1,106.69 

1,032.31 

74.38 


(156.17) 

(155.58) 


Seniors 

1,126.95 

1,062.94 

64.01 


(155.61) 

(155.09) 


URBAN 




First-year students 

1,064.54 

996.18 

68.36 


(145.32) 

(141.83) 


Seniors 

1,077.36 

1,008.52 

68.84 


(161.65) 

(144.33) 



Note: This table includes means and standard deviations (in parentheses). 
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Figure 2. Gaps in CLA Performance between First-Generation and Non-First 
Generation Students, Controlling for Individual Characteristics 
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*p<0.01, *p<0.05 (None of the reported differences is statistically significant.) 


Figure 3. Gaps in CLA Performance between First-Generation 
and Non-First Generation Students by Urbanicity, 
Controlling for Individual Characteristics 


Urban 


Figure 2, which displays the 
results graphically, shows no 
statistically significant gaps 
between first-generation and non- 
first-generation students in either 
first or senior year, after adjusting 
for individual-level differences. 3 
The gaps between first-generation 
and non-first-generation students 
in Figure 2 are also of small 
magnitude, often just a fraction 
of the gaps observed in the 
descriptive data. Though there is 
some variation in the magnitude 
of the gaps across different 
components of the CLA, the 
overall pattern of no large or 
statistically significant difference 
between first-generation and non- 
first-generation students holds for 
all outcomes examined. 

Although the overall performance 
of first-generation students does 
not differ from that of their 
non-first-generation peers, net of 
controls, these averages may hide 
variation in performance across 
different geographical contexts. 

To consider this possibility, the report turns to examining CLA performance of first-generation and non-first- 
generation students in urban and non-urban settings. The descriptive results in Table 1 show that the gap in 
pre-college academic preparation between first-generation and non-first-generation students is similar across those 
two contexts. This would imply that the gap in CLA performance is likely to be similar in urban and non-urban 
settings. To evaluate this claim statistically, Table 3A in the appendix shows results from regression analyses 
estimating gaps in CLA performance between first-generation and non-first-generation students, after adjusting 
for pre-college academic preparation and sociodemographic characteristics. Figure 3 graphically illustrates 
selected findings. 
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Note: **p<0.01, *p<0.05 (None of the reported differences is statistically significant.) 


The y-axis range in Figure 2 (and all similar figures reporting gaps in CLA performance) is approximately 1/2 standard deviation. 
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Results from regression analyses Table 2. Overlap between Different Definitions of Underrepresented Status 

reveal no statistically significant 
gaps in CLA performance between 
students from first-generation and 
non-first-generation backgrounds, 
net of controls. This pattern of no 
significant difference holds for both 
first-year students and seniors and in 
both urban and non-urban settings. 

There is one exception to this pattern, 
which is a statistically significant gap 
among first-year students in the “make 
an argument” component of the CLA 
(see Table 3A in the appendix). Although 
statistically significant, that gap is 
small in magnitude. Thus, though there 
is some variability in results, which is 
to be expected given smaller sample 
sizes resulting from dividing students 
into categories based on both their first-generation status and urbanicity (the degree to which a location is urban), 
the overall pattern of no difference in CLA performance between first-generation and non-first-generation 
students is reasonably consistent across outcomes. These results imply that first-generation students perform 
equally well on the CLA as their non-first-generation peers after adjusting for individual-level differences between 
the two groups. 

The next two sections of the report, III and IV, will examine CLA performance by Pell-grant eligibility and 
race/ethnicity. Before presenting those results, it is worthwhile to note that there is a substantial amount of 
overlap between students in different underrepresented categories. As Table 2 shows, among first-generation 
first-year students, approximately one-half are Pell-eligible and one-quarter are African-American and Hispanic. 
The proportions of Pell-eligible and African-American/Hispanic students are much smaller among non-first- 
generation first-year students. This table also reveals that the senior sample includes a slightly smaller proportion 
of first-generation and Pell-eligible students and a substantially smaller proportion of African-American and 
Hispanic students. 



First-Year Students 

Seniors 

TOTAL 

First-generation 

53% 

48% 

Pell-eligible 

41% 

36% 

African-American/Hispanic* 

19% 

12% 

FIRST-GENERATION 

Pell-eligible 

53% 

49% 

African-American/Hispanic* 

26% 

15% 

NON-FIRST-GENERATION 

Pell-eligible 

27% 

24% 

African-American/Hispanic* 

11% 

8% 


For this analysis, African-American and Hispanic students are compared with white students ; 
students of other racial/ethnic backgrounds are excluded from analysis. 
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III. CL A Performance of Students by Pell- Grant Eligibility 


The discussion of CLA performance for Pell- 
eligible and non-Pell-eligible students begins 
with descriptive results. Figure 4a shows that 
Pell- eligible students enter higher education 
with lower levels of skills as measured by 
the CLA. Pell-eligible first-year students 
have lower performance on all components 
of the CLA compared to non-Pell-eligible 
first-year students. Yet though this difference 
is statistically significant, it is not of large 
magnitude ( see Table 4A in the appendix 
for effect size estimates). Among seniors, 
descriptive results suggest that Pell-eligible 
and non-Pell-eligible students perform equally 
well on different components of the CLA (see 
Figure 4b). Differences between Pell-eligible 
and non-Pell-eligible seniors are typically 
about half of the differences among first-year 
students. None of the differences in the senior 
sample is statistically significant. 

The descriptive results, which show gaps in 
CLA performance among first-year students 
but not seniors, seem to imply that Pell- 
eligible students catch up with their non-Pell- 
eligible peers during their time in college. 

The following analyses, however, indicate 
that conclusion to be premature. Figures 4a 
and 4b present descriptive statistics from 
cross-sectional samples. These figures do not 
adjust for differences between Pell-eligible 


Figure 4a. First-Year Student CLA Scores, by Pell-Grant Eligibility 
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task** argument** argument** 
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Note: **p<0.01, *p<0.05 


Figure 4b. Senior CLA Scores, by Pell-Grant Eligibility 
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and non-Pell- eligible students on various individual 
characteristics, and they do not include the same 
students in both first-year and senior samples. 

Table 3 illustrates patterns for one important 
characteristic: pre-college academic performance. 
Among first-year students, the difference between 
the two groups is as expected: Pell-eligible students 
have lower pre-college academic performance than 
non-Pell-eligible students. The gap is still present in 
the senior sample, but it is smaller. Both Pell-eligible 
and non-Pell-eligible students have higher pre-college 
academic preparation in the senior sample than in 
the first-year student sample. Though that is to be 
expected in a cross-sectional sample, it is notable 
that the difference in academic preparation between 
first-year and senior samples is twice as large for 
Pell-eligible students as for non-Pell-eligible students. 
Table 3 thus reveals a convergence in academic 
preparation between Pell-eligible and non-Pell- 
eligible samples in the senior year. Since academic 
preparation is related to CLA performance, this 
pattern would imply convergence in 
CLA performance, which is exactly 
what was observed in Figure 4b. 


Table 3. Students’ Pre-College Academic Performance, 
by Pell-Grant Eligibility and Urbanicity 



Non-Pell 

Pell 

Difference 

TOTAL 

First-year students 

1075.30 

(154.42) 

1,014.84 

(151.96) 

60.46 

Seniors 

1,089.67 

(158.99) 

1043.80 

(158.20) 

45.87 

NON-URBAN 

First-year students 

1,085.17 

(158.03) 

1,044.95 

(160.78) 

40.22 

Seniors 

1,117.63 

(155.74) 

1,055.87 

(156.48) 

61.76 

URBAN 

First-year students 

1,060.49 

(147.70) 

983.46 

(135.35) 

77.03 

Seniors 

1,050.75 

(155.35) 

1,032.28 

(159.17) 

18.47 


Note: This table includes means and standard deviations (in parentheses). 

Figure 5. Gaps in CLA Performance between Pell-Eligible and 
Non-Pell-Eligible Students, Controlling for Individual Characteristics 


Performance task 


Argument 


Make an argument Break an argument 


To examine whether CLA 
performance differs by Pell- 
grant eligibility, after adjusting 
for individual differences, the 
report includes regression analyses 
of different components of the 
CLA, controlling for students’ 
sociodemographic characteristics 
(gender, race/ethnicity, parental 
education, and an indicator of English 
as a primary language) and pre-college academic preparation ( see Table 5A in the appendix for more details and 
alternative model specifications). Figure 5, which displays the results graphically, shows negligible or non-existent 
gaps in CLA performance between Pell-eligible and non-Pell-eligible students in both the first year and senior 



Note: **p<0.01, *p<0.0 5 (None of the reported differences is statistically significant.) 
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year, net of controls. None of the 
gaps is either statistically significant 
or of large magnitude. The apparent 
differences in the descriptive results 
thus reflect variation in individual- 
level characteristics. 

Although Pell-eligible and non- 
Pell-eligible students perform 
equally well on different components 
of the CLA, after adjusting for 
individual-level characteristics, it is 
important to consider whether the 
same pattern holds in both urban 
and non-urban settings. Examining 
CLA performance in urban and non-urban settings is complicated by patterns revealed in Table 3. Students in 
non-urban settings are more academically prepared than students in urban settings. This pattern holds for both 
Pell-eligible and non-Pell-eligible students, as well as for first-year and senior samples. But beneath that simple 
observation lies a complicated set of patterns. In urban settings, Pell-eligible first-year students lag substantially 
behind their non-Pell-eligible peers in pre-college academic preparation. This difference is substantially smaller 
in the senior year — indeed it is only one-quarter of the size in the first year. In urban settings, thus, Pell-eligible 
students are much less academically prepared than their non-Pell-eligible peers in the first-year sample but are 
relatively close to the academic preparation of their non-Pell-eligible peers in the senior sample. The pattern is 
reversed in non-urban settings, where the gap in academic preparation between Pell-eligible and non-Pell-eligible 
students is larger in the senior year than in the first year. Any consideration of CLA performance of Pell- eligible 
and non-Pell-eligible students in urban and non-urban settings thus must attend to differences in academic 
preparation. 

Figure 6 shows gaps in CLA performance between Pell-eligible and non-Pell-eligible students in urban and 
non-urban settings, controlling for students’ sociodemographic characteristics and pre-college academic 
performance (see Table 6A in the appendix for complete results and alternative model specifications). After adjusting 
for individual-level characteristics, there are no statistically significant differences in CLA performance between 
Pell-eligible and non-Pell-eligible students; this finding holds for first-year students and seniors in both urban 
and non-urban settings. The estimates for these models are less stable given relatively small sample sizes for each 
sub-group examined. Consequently, there is more variability across outcomes, particularly in urban settings. The 
overall pattern, however, is one of no difference in CLA performance between Pell-eligible and non-Pell-eligible 
students, net of controls. 


Figure 6. Gaps in CLA Performance between Pell-Eligible and Non-Pell-Eligible 
Students by Urbanicity, Controlling for Individual Characteristics 
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IV. CLA Performance by Race/Ethnicity 


Figure 7a. First-year Student CLA Scores, by Race/Ethnicity 
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Figure 7b. Senior CLA Scores, by Race/ Ethnicity 


Although students from racial/ethnic minority 
groups, and African-American and Hispanic 
students in particular, are more likely to be 
first-generation as well as Pell- eligible, it is 
worthwhile to examine their performance 
independently. Figures 7a and 7b show 
descriptive results for different components of 
the CLA for African-American and Hispanic 
students in comparison to white students. 

Descriptive results indicate that African- 
American and Hispanic students enter college 
performing substantially below their white peers 
on all CLA components. These differences 
are not only statistically significant but also of 
greater magnitude than any of the gaps reported 
previously for either Pell-eligible or first- 
generation students (see Table 7 A in the appendix 
for effect size estimates). These patterns reveal 
a distinct disadvantage faced by racial/ethnic 
minority students entering higher education. 

CLA performance of seniors presents a mixed 
picture (see Figure 7b). The gap between African- 
American/Hispanic and white students for 
the break an argument component is of small 
magnitude and not statistically significant. 

Gaps for the other components of the CLA 
remain statistically significant, although of 
varied magnitudes. The largest gap is for the 

performance task, where African-American/Hispanic seniors perform 0.65 standard deviation below their white 
peers. Indeed the difference between African-American/Hispanic and white students on the performance task 
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is larger in the senior sample than in 
the first-year sample. Some of this 
variability in the results may reflect 
relatively small sample sizes for 
African-American/Hispanic students 
in the senior year (approximately 140 
for performance task and 120 for 
other components of the CLA; see 
Table 7A). 

Making comparisons between 
first-year and senior samples is 
further complicated by the ethnicity 
distribution patterns noted in Table 
2: While the first-year student sample includes 19 percent of African-American/Hispanic students, the senior 
sample includes only 12 percent of students in these racial/ethnic groups. The difference between the first and 
senior year is particularly pronounced for first-generation students: Among first-generation students, 26 percent of 
first-year students and only 15 percent of seniors are African-American/Hispanic. First-year and senior samples 
thus seem to include different groups of African-American/Hispanic students. 

African-American/Hispanic students differ from their white peers not only with respect to first-generation 
status but a range of other background characteristics, including academic preparation. An important question 
thus is whether gaps in CLA performance between African-American/Hispanic and white students persist after 
controls. Table 8A in the appendix reports results from regression models predicting each of the components of 
the CLA while controlling for students’ sociodemographic characteristics (gender, parental education, Pell-grant 
eligibility, and an indicator of English as a primary language) and pre-college academic preparation. Figure 8 
summarizes the results by showing adjusted gaps in CLA performance between African-American/Hispanic and 
white students. This figure shows no statistically significant gaps between African-American/Hispanic and white 
students in either first-year or senior samples. Some differences in magnitude exist, with the gap in performance 
task in the senior year being particularly pronounced. The overall pattern, however, indicates no racial/ 
ethnic differences in CLA performance in either the first or senior year after controls for pre-college academic 
performance and background characteristics. 

Although it would be valuable to consider whether the observed racial/ethnic patterns in CLA performance are 
the same in urban and non-urban settings, this comparison is not possible with the available data. As noted in 
Table 7A, the number of African-American/Hispanic students is already quite low in the senior sample. Further 
disaggregating this sample by urbanicity would produce too small a sample to render reliable estimates. 


Figure 8. Gaps in CLA Performance between African-American/Hispanic and 
White Students, Controlling for Individual Characteristics 
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V. CLA Performance in Urban and Non-Urban Settings 


Descriptive Differences and Regression Results for Urban and Non-Urban Settings 
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Figure 9b. Senior CLA Scores, by Urbanicity 


Although differences between urban and Figure 9a. First-Year Student CLA Scores, by Urbanicity 

non-urban contexts have been considered 
throughout the report, the analyses heretofore 
focused on examining whether gaps between 
certain groups of students vary in urban and 
non-urban settings. At this point, the report 
turns to a closer examination of urban vs. 
non-urban differences. The report will begin 
by treating urbanicity as a characteristic 
of students and thus reporting individual- 
level analyses akin to those performed for 
first-generation status, Pell-grant eligibility, 
and race/ethnicity. These analyses present 
a broader view of urban vs. non-urban 
differences and depend less on characteristics 
of specific samples at each institution. 

Moreover, they allow for estimation of 
regression analyses using all of the available 
data, thus adjusting CLA estimates for a 
range of students’ background characteristics. 

Descriptive results in Figure 9a reveal that 
students in urban and non-urban settings 
perform equally well on different components 
of the CLA at entry into higher education. 

Differences among first-year students are 
very small and not statistically significant (see 
Table 9A in the appendix for complete results 

and effect size estimates). Senior-year results are slightly more mixed (see Figure 9b). Gaps between seniors in urban 
and non-urban settings are small in magnitude and not statistically significant for argument components of the 
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CLA. The difference in the performance task, however, is 0.43 standard deviations, a moderate magnitude and 
statistically significant. 


Notwithstanding the difference in the senior-year performance task measure, it is notable that descriptive results 
reveal little variation in CLA performance across urban and non-urban settings. As students across those two 
settings differ from each other along multiple dimensions, it maybe anticipated that the regression analyses will 
even further reduce the relatively small gaps observed in Figures 9a and 9b. Table 10A in the appendix shows 
results from regression analyses of CLA performance, adjusted for students’ sociodemographic characteristics 
(gender, race/ethnicity, Pell-grant eligibility, parental education, and an indicator of English as a primary 
language) and pre-college academic preparation. Figure 10 displays selected results graphically. 


Figure 10. Gaps in CLA Performance between Students in Urban 
and Non-Urban Areas, Controlling for Individual Characteristics 
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Patterns in Figure 10 reveal no notable 
or statistically significant differences 
in CLA performance between first- 
year students in urban and non-urban 
settings. The same pattern holds for the 
senior year, except for the performance 
task component of the CLA. 

Controlling for individual characteristics 
reduces, but does not eliminate, the gap 
in performance task between seniors 
in urban and non-urban settings. This 
one exception deserves more careful 
exploration in future research, but given 
that the other components of the CLA show no difference between the two settings, the weight of the evidence 
implies that there is no difference in CLA performance between urban and non-urban settings. It is important to 
note that these findings are not simply an artifact of using the full sample as opposed to the representative sample 
(see the Methodological Appendix for discussion of sample differences). If the representative sample were used, 
all of the reported patterns would hold, with the gaps differing by only a few points (e.g., for the performance 
task, the gap is 53 points in the full sample and 49 points in the representative sample after controlling for student 
characteristics). 
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Institutional CLA Performance in Urban and Non-Urban Settings 

Though the preceding analyses treated urbanicity as a characteristic of students, this project was designed with 
urbanicity as an institution-level attribute. At this point, the report thus shifts to focus on institutions as units 
of analysis. This shift necessitates several changes in perspective. The first one is focusing on data from the 
representative sample as opposed to the full sample, as was the case for preceding analyses. Each institution aimed 
to collect information from a representative sample of students and an in-depth sample of Pell-eligible and/or 
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first-generation students. Although institutions were differentially successful in these endeavors and differentially 
coded students as being in the representative sample or the in-depth sample, claims about individual institutions 
will focus on what they designated as the representative sample (for a more detailed discussion of the two samples, 
see the Methodological Appendix). Second, focusing on institutions shifts the analytic frame to the value-added 
approach, which emphasizes an institution’s contribution to the development of higher-order skills. 


Figure 11 shows descriptive patterns for the performance task of the CLA for first-year students and seniors at 
each institution. To protect the confidentiality of the institutions, they are given generic names (college 1, college 
2, etc.) and separated into urban and non-urban settings. It is important to note that while all institutions are 
shown in the figure, the reliability of estimates varies across institutions, and some of the estimates are based on a 
very small number of cases (see sample size for each institution in Table 11A and Table 12A in the appendix). Estimates 
for college 9 in particular are based on a very small number of cases (fewer than 20 cases in each student cohort). 


The light blue portion of the bars 
indicates average performance by 
first-year students in each institution. 
The dark blue portion of the bars 
represents the difference between 
the average first-year and the average 
senior score. These descriptives reveal 
much variation across institutions in 
both first-year and senior samples. 

On average, the dark blue portion 
of the bars is larger for non-urban 
institutions. This implies that the 
difference between senior and first- 
year scores is larger in non-urban 
than in urban settings. There is a 
substantial amount of variation within 
both settings, however. Moreover, 
these findings need to be interpreted 
with caution given that they are based 
on cross-sectional data. 

The patterns for other components of 
the CLA are more similar across the 
two settings. Figure 12 shows average 
scores for the argument component of 
the CLA for first-year students and 


Figure 11. CLA Performance Task Scores for 
First-Year Students and Seniors, by Institution 
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Figure 12. CLA Argument Scores for 
First-Year Students and Seniors, by Institution 
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seniors at each institution. In this figure, there is no discernible pattern that would differentiate urban from non- 
urban institutions. There is a substantial amount of variation across institutions within both urban and non-urban 
contexts and no clear difference between institutions in the two settings. This pattern would be anticipated from 
previously reported individual-level regression analyses, which reported no difference between urban and non- 
urban settings with respect to student performance on the argument component of the CLA. 

Figures 11 and 12 show descriptive patterns without any adjustment for student characteristics. To consider 
whether institutional performance differs after adjusting for student characteristics, the report turns to 
institutional value-added scores. Value-added scores used in this portion of the report are computed by 
the Council for Aid to Education (CAE) using representative samples for each institution. An institution’s 
value-added score indicates the degree to which the observed senior mean CLA score meets, exceeds, or falls 
below expectations. The CAE calculates value-added scores by controlling for students’ pre-college academic 
performance. 


Before reporting value-added estimates, it is worthwhile to reflect on the question of whether pre-college 
academic performance represents an adequate control. Though the report cannot answer this question definitively, 
appendix tables provide some insights. All regression analyses in the appendix include two different models: one 
controlling only for students’ pre-college academic preparation and the other one controlling for students’ pre- 
college academic preparation and several other background characteristics. Examination of the results indicates 
that controlling for pre-college academic performance substantially reduces gaps between all of the groups 
examined (first-generation vs. non-first-generation, Pell-eligible vs. non-Pell- eligible, African-American/Hispanic 
vs. white). And notably, pre-college 


academic performance alone reduces 
the gaps in CLA performance as much 
as all of the other sociodemographic 
characteristics considered jointly. Pre- 
college academic performance is thus the 
single most important factor to be used in 
the adjustment of CLA performance for 
pre-existing student differences. 


Figure 13. Value-Added CLA Performance Task Scores by Institution 


2.5 


1.5 


0.5 


♦ C3 « C6 


♦ C4 


O C13 
O C12 
O Cll 


O C16 


O C18 


O S19 


O C15 


♦ C9 


O C17 


Figure 13 reports standardized value- 
added scores for the performance task 
of the CLA for each institution. This 
figure reveals two notable findings. First, 
on average, non-urban institutions have 
slightly higher value-added scores on the 
performance task of the CLA — most 
of the non-urban institutions are in the 
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positive range of value-added scores, with 
only two non-urban institutions having 
negative value-added scores. Second, 
a substantial amount of variation in 
value-added scores occurs within each of 
those settings. Some urban institutions 
have positive value-added scores and 
some have negative value-added scores. 

The same pattern holds for non-urban 
institutions. And indeed, most of the 
institutions in urban and non-urban 
settings have value-added scores within a 
close range. 

Figure 14 shows institutional value-added 
scores for the argument component of 
the CLA. As would be anticipated from 
the descriptive results, this figure shows 
much more variation within each setting and no clear pattern differentiating urban from non-urban institutions. 
Some institutions in both settings have positive value-added scores, and some institutions in both settings have 
negative value-added scores. There is no clear tendency for institutions in either the urban or non-urban setting to 
cluster in a particular section of the graph. Findings in Figures 13 and 14 highlight the challenges of identifying 
whether contextual factors, such as urban vs. non-urban settings, are related to CLA performance. As is clear 
from these figures, the predominant variation is within each setting not across it. Even in instances where small 
differences between urban and non-urban contexts (e.g., the performance task) may exist, those differences are 
overshadowed by variation within urban and non-urban contexts. Both urban and non-urban settings have high 
and low-performing institutions. 

Institutional CLA Value-Added Scores Assigned to Performance Levels 

In addition to reporting value-added scores for each institution, the CAE assigns institutions to different 
performance levels. These performance levels are based on standardized value-added scores (i.e., value-added 
scores expressed in standard deviations such as those reported in Figures 13 and 14) as follows: Institutions that 
fall between -1.00 and +1.00 are classified as “near expected,” between +1.00 and +2.00 are “above expected,” 
between -1.00 and -2.00 are “below expected,” above +2.00 are “well above expected,” and below -2.00 are “well 
below expected.” To simplify graphic representations in this section, the categories are collapsed into three levels: 
“above expected” (including “well-above expected”), “near expected,” and “below expected” (including “well-below 
expected”). Most of the institutions in the sample fall in the “near expected” category across different measures. A 
few of the institutions are classified as “above expected” or “below expected,” with only one institution falling into 


Figure 14. Value-Added CLA Argument Scores by Institution 
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the “well-below expected” category 
for the argument component of the 
CLA (see Table 13 A in the Appendix). 

Figures 15 and 16 include 
previously reported value-added 
scores, with two horizontal lines 
separating above expected, near 
expected, and below expected 
categories. These figures show that 
most of the institutions in both 
urban and non-urban settings are 
in the near-expected category. 

Four non-urban institutions are 
in the above-expected category 
for performance task, without any 
urban institutions being classified 
in this category. This would seem to 
suggest that non-urban institutions 
perform slightly better. Institutions 
in the above-expected portion of 
the graph are quite close to the 
line, however, and thus are quite 
close to being in the near-expected 
category. Moreover, non-urban 
institutions do not perform as well 
when considering the argument 
component of the CLA. Only one 
non-urban institution, along with 
two urban institutions, is in the 
above-expected category for the 
argument component of the CLA. 

Two non-urban institutions (and no 
urban institution) are in the below- 
expected category for the argument 
component of the CLA. As noted 
previously, thus, there is more 
variation in performance within 
urban and non-urban settings than across them. 


Figure 15. Value-Added CLA Performance Task Scores by Institution, 
Divided into Performance-Level Categories 
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Figure 16. Value-Added CLA Argument Scores by Institution, 
Divided into Performance-Level Categories 
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Another approach to considering variation within urban and non-urban settings is to examine students’ individual 
performance. In addition to classifying institutions into specific performance categories, the CAE creates 
expected scores for students. Students’ expected CLA scores are based on a regression model adjusting for their 
academic preparation as well as average CLA scores and academic preparation of the institution they attend. 

Based on the regression equation, students’ expected scores are compared to their observed scores and resulting 
deviation scores (or residuals) are divided into quintiles, with students in the top quintile designated as “well- 
above expected,” second quintile as “above expected,” middle quintile as “near expected,” fourth quintile as “below 
expected,” and the bottom quintile as “well-below expected.” 


Figure 17. Percentage of Seniors Performing at 
Various Levels in Urban and Non-Urban Settings 
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a disproportionate number of students who 
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and non-urban settings. For the argument component of the CLA, the distribution of student performance in 
urban and non-urban settings is virtually identical. There are small differences for the performance task, as would 
be expected from previous analyses, showing that students in non-urban settings seem to perform slightly better 
on the performance task component of the CLA. Despite this variation, similarities across the two settings are 
much more pronounced than the differences. 
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Moreover, it may be valuable to note that high and low-performing students are found at all institutions. One 
straightforward way to illustrate this point is to examine student performance at institutions of varying levels of 
performance. It is important to keep in mind that institutional performance levels are based on absolute cut-offs, 
while student performance levels are based on a relative location in the distribution (i.e., quintiles). The two scales 
are thus not perfectly matched, but they are adequate for the purposes of this illustration. 

Institutions in the near-expected category are closest to the average and thus should have approximately 40 
percent of students who perform in the above/well-above expected category, 20 percent in the near-expected 
category, and 40 percent in the below/well-below average category. That is indeed what is observed in Figures 
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18 and 19. Moreover, institutions that are 
in the above-expected category have more 
than 40 percent of students in the above/ 
well-above category, and institutions in the 
below-expected category have more than 40 
percent of students in the below/well-below 
category. This is precisely what would be 
expected based on the definitions. 

It is worthwhile to note, however, that, 
regardless of the overall institutional 
performance, all of these institutions enroll 
students of varying degree of ability. Even 
institutions classified as above expected 
have some students who perform below 
what would be expected. Similarly, even 
institutions classified as below expected have 
a substantial proportion of students who 
perform above what would be expected. As 
Charles Blaich, director of inquiry in the 
liberal arts at Wabash College, often notes, 
every institution has a “zone of excellence” 
and a “zone of despair.” Every institution has 
students who perform above expectations 
as well as those who perform below 
expectations. Institutional leaders thus face a 
unique challenge of identifying the students 
who perform well and the ones who seem to 
be losing ground (i.e., performing at lower 
levels than expected) and developing specific 
strategies to facilitate student learning on 
their campuses. 


Figure 18. Percentage of Seniors Performing Below-Near-Above 
Expected on the Performance Task Component of the CLA, 
at Institutions of Varying Levels of Performance 
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Figure 19. Percentage of Seniors Performing Below-Near-Above 
Expected on the Argument Component of the CLA, at Institutions 
of Varying Levels of Performance 
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Though lacking longitudinal data and therefore the ability to make direct claims about growth over time, findings 
presented are consistent with the following observations. Independent urban and non-urban institutions do 
equally well in educating the students they enroll (as assessed by the CLA). Independent urban and non-urban 
institutions do equally well in educating different groups of students (as assessed by the CLA), including students 
from underrepresented groups. 
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VI. Conclusion 


This study examined gaps in CLA performance between several underrepresented groups of students and their 
more advantaged peers. In particular, the study compared first-generation and non-first-generation students, 
Pell-eligible and no n-Pell- eligible students, and African-American/Hispanic and white students. In addition to 
reporting average CLA performance for those groups, the report also considered whether student performance 
varied across urban vs. non-urban settings. 

Three main findings emerge from presented analyses: 

■ After controlling for individual-level characteristics, and particularly academic preparation, there are no 
differences in CLA performance among different groups of students. This finding holds in both urban and 
non-urban institutional settings. 

* There is substantially more variation within urban and non-urban settings than across those two settings. 
Individual institutions, not broad categories such as urban and non-urban, present a more productive focus 
of analysis and policy development. 

■ Independent urban and non-urban institutions do equally well in educating the students they enroll (as 
assessed by the CLA). They also do equally well in educating different groups of students, including 
students from underrepresented groups. 

These findings have several notable implications: 

■ The most important predictor of CLA performance is academic preparation. This is also the primary factor 
that explains inequalities in CLA performance among different groups of students. This finding suggests 
that postsecondary institutions should work with local high schools to improve student preparation or 
influence broader state and national conversations to prepare high school students for college-level work. 

■ There is more variation in CLA performance within specific institutional settings (e.g., urban and 
non-urban) than across those settings. Analyses and policy interventions thus need to focus on specific 
institutions, not broad categories of institutions. 
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■ All institutions have students who perform at different levels, producing much more variation within 
institutions than across them. Institutions would thus benefit from focusing within — studying carefully 
their own students and understanding the groups of students that perform well and how their successes can 
be replicated, as well as the groups of students that do not perform as well and the specific strategies that 
are needed to improve their outcomes. 

Finally, this report could not have been written without the concerted effort of many institutions. Moreover, 
institutions cannot improve their outcomes without understanding the challenges and successes of their students. 
To better examine variation on student learning outcomes within an institution, additional assessment measures 
beyond those examined in this study will be needed. Investing in institutional infrastructure to collect high- 
quality assessment data is thus an important part of the puzzle of improving learning outcomes for all students. 

It is important to note that the results presented in this report are based on cross-sectional data including 
samples of first-year students in 2010 and seniors in 2011. Moreover, the report includes two different samples: 
a representative sample of the student population and an in-depth sample of first-generation and low-income 
students. All of the individual-level analyses are based on the full sample (including the representative sample and 
the in-depth sample), while institution-level analyses are based on the representative sample only. Comparisons 
of those samples in both the first and senior years are discussed in the Methodological Appendix. Discussion 
of different samples illuminates the challenges of data collection efforts and subsequent statistical analyses. 
Supporting the development of stronger institutional infrastructure for data collection efforts would facilitate 
more robust analyses in the future. 
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Methodological Appendix 


Overall Sample Definition 

This report is based on the fall 2010 first-year cohort and spring 2011 senior cohort at 19 participating institutions, 
all small and mid-sized private, nonprofit colleges and universities. Students with missing information on the key 
variable of interest, Pell status, which was missing for six students, were excluded from analyses. This restriction 
produces the beginning sample size of 3,479 first-year students and 2,440 seniors. This overall sample size 
includes two distinct samples: a representative sample and an in-depth sample of first-generation and low-income 
students. (Though race/ethnicity is examined in this report, institutions were not asked to provide an in-depth 
sample along this dimension.) The representative sample includes 2,645 first-year students and 1,999 seniors. The 
in-depth sample includes 834 first-year students and 441 seniors. Comparison of the two samples is discussed 
under the heading of “sample comparisons.” It is important to note that approximately half of the students took 
the performance task component of the Collegiate Learning Assessment (CLA) and the other half took the 
argument components of the CLA. Therefore, reported analyses for each component of the CLA are based on 
approximately half of the sample. Appendix tables include sample sizes for each outcome and group examined. 

Variable Definitions 

The report focuses on exploring CLA performance of underrepresented groups of students in comparison to their 
more advantaged peers. Underrepresented status is defined along three different, albeit overlapping, dimensions: 
first-generation status, Pell-grant eligibility, and race/ethnicity. For the purposes of selecting an in-depth sample, 
a “first-generation” student was defined as a student whose parent(s) did not attend any college, and a “low- 
income” student was defined as a student who was a Pell grant recipient or who was eligible to receive a Pell grant. 
Pell-grant eligibility is a variable denoting whether a student received or was eligible for a Pell grant. Even with 
over- sampling, however, the proportion of students whose parents had no college experience ended up being quite 
small. Approximately one-quarter of first-year students and less than 20 percent of seniors in the full sample came 
from families where parents had no college experience. As only approximately half of the students completed the 
performance task, and the other half completed the argument components of the CLA, each of those samples was 
split in half, substantially decreasing the reliability of results. For the purposes of the analyses presented in the 
report, a first-generation student thus is defined as a student coming from a family where neither parent completed 
a college degree. Approximately one-half of first-year students and seniors had at least one parent who completed 
college or graduate/professional degrees — therefore, approximately one-half of the sample is designated as non- 
first-generation in this study. 
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The third aspect of underrepresented status is race/ethnicity. The original variable included six categories, but most 
of them, except for white, had a relatively small number of cases. In this report, African-American and Hispanic 
students are combined into one category and compared to white students. Other racial/ethnic groups are omitted 
from analyses of racial/ethnic differences in CLA performance. Including all non-white groups into one category 
produces too much heterogeneity. Different racial/ethnic groups tend to have varied levels of pre-college academic 
performance (and subsequent CLA performance); some perform relatively similarly to white students and some do 
not. African-American and Hispanic students are most similar along these dimensions. They are combined into 
one category because each group includes too few students to provide reliable estimates, particularly in the senior 
sample. 

The report explores variation in student performance between urban and non-urban settings. “Urban” was 
defined as a city with a population greater than 100,000 or a large suburb located in close proximity to a major 
metropolitan area where the institution serves a high percentage of students from that metropolitan area. Among 
19 institutions participating in the study, nine were located in urban settings and ten in non-urban settings. 

The key outcome of interest is students’ performance on the CLA, a standardized assessment which aims to 
capture higher-order collegiate skills, including critical thinking, complex reasoning, and written communication. 
The CLA provides four different scores: performance task, make an argument, break an argument, and argument 
(which is the average of make an argument and break an argument scores). Throughout most of the report, results 
for all four CLA components are reported. In general, the patterns are consistent across measures, increasing 
confidence in the reported results. 

Analysis 

For each dimension of underrepresented status, the report first presents descriptive results. Average CLA 
performance of students in underrepresented groups is compared to the average CLA performance of their more 
advantaged peers. The comparisons include: first-generation vs. non-first-generation, Pell-eligible vs. non-Pell- 
eligible, and African-American/Hispanic vs. white. 

Although these descriptive results provide valuable information, they are difficult to interpret since students from 
underrepresented groups differ from students who are not in those groups. The report thus includes regression 
analyses of CLA performance, controlling for specific individual-level characteristics. The strongest predictor 
of CLA performance is students’ pre-college academic preparation (i.e., entering academic ability score: SAT 
Math plus Verbal, ACT Composite, or Scholastic Level Exam [SLE] scores on a common scale). The report 
thus includes two sets of models controlling for: a) pre-college academic preparation and b) pre-college academic 
preparation and sociodemographic characteristics. 

All descriptive results are based only on non-missing data. To preclude deleting cases from regression analyses, 
however, a mean substitution method was used (i.e., mean is substituted for missing data and a dummy indicator 
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denoting that the substitution is made is added to the models). The number of missing cases is quite small, but 
given that some of the analyses already rely on a small number of cases, preserving cases is valuable. Four percent 
of cases were missing pre-college academic performance, 0.25 percent gender, and 2 percent race/ethnicity. Race/ 
ethnicity is not mean substituted in the models focusing on African-American/Hispanic vs. white comparisons. 

All regression analyses and statistical tests are adjusted for clustering of students within institutions. 

Sample Comparisons 

This study includes two distinct samples: the representative sample and the in-depth sample of first-generation and 
low-income students. The in-depth sample was added to increase the reliability of estimates for underrepresented 
groups of students. Individual-level regression analyses were performed on the full sample (i.e., including both the 
representative and the in-depth sample) to maximize the number of cases for each group of interest. Institution- 
level analyses were based on the representative sample. The Council for Aid to Education (CAE) also used the 
representative sample to estimate institutional value-added scores. The CIC and CAE representative samples differ 
slightly because CAE deleted cases without valid pre-college academic preparation. Tables 11 A and 12A show the 
distribution of cases and mean scores for the performance task component of the CLA for each institution across 
different samples. It is notable that many institutions do not have an in-depth sample; thus, the representative and 
the full sample are the same. 

When considering whether to use the representative or the full sample, in addition to the concern about the number 
of cases available for specific groups, it is worthwhile to note that the representative sample is not always necessarily 
representative. For example, three institutions in the sample included all of their Pell-eligible students in the in- 
depth sample, leaving none in the representative sample. There was also variation in the ability of institutions to 
obtain representative samples. For example, a supplementary comparison was conducted comparing the percentage 
of white and Pell-grant eligible students in two samples: the representative sample in this dataset and institutional 
data reported in the Integrated Postsecondary Education Data System (IPEDS) for 2009. Those comparisons reveal 
variation in the extent to which representative samples in this study mimic data reported in IPEDS. 

For all of the aforementioned reasons, regression analyses in this report are based on the full sample. 

Supplementary analyses, however, were conducted to compare the sensitivity of results to the sample used. CLA 
performance of first-year students and seniors in the full sample, in-depth sample, and representative sample was 
compared. The comparison focused on Pell-eligible vs. non-Pell-eligible students since those two groups show 
the most variation. After calculating the CLA performance for each sample, the CLA performance was weighted 
based on the representation of Pell-eligible and non-Pell-eligible students in each of the samples. The results from 
the weighted data were closer to the results for the full sample than the results for the representative sample. In 
addition, regression analyses were conducted on both the full and the representative sample. Regression results for 
the two samples are substantively identical — i.e., neither sample shows gaps in CLA performance between 
different groups after controlling for individual-level characteristics. Reliance on the full sample thus does not 
appear to bias the reported results. 
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Appendix Tables 


Table 1A. CLA Scores, by First- Generation Status 

First-generation Non-first generation Mean difference Mean difference 



N 

Mean 

SD 

N 

Mean 

SD 

Raw scores 

Effect size (SD) 

First-year students 









Performance task 

925 

1,021.06 

158.58 

824 

1,066.50 

163.36 

-45.44** 

-0.28 

Argument 

901 

1,024.31 

134.49 

791 

1,064.88 

136.72 

-40.57** 

-0.30 

Make an argument 

917 

1,016.84 

157.58 

799 

1,063.86 

152.95 

-47.02** 

-0.30 

Break an argument 

908 

1,028.45 

156.07 

795 

1,065.24 

157.70 

-36.79** 

-0.23 


Seniors 


Performance task 

622 

1,161.05 

173.39 

630 

1,187.91 

170.69 

-26.86* 

-0.16 

Argument 

537 

1,146.35 

139.32 

633 

1,169.62 

143.84 

-23.27* 

-0.16 

Make an argument 

542 

1,120.98 

160.80 

636 

1,151.58 

166.96 

-30.60** 

-0.19 

Break an argument 

538 

1,172.09 

158.65 

637 

1,184.78 

166.12 

-12.69 

-0.08 


Note: **p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 


Table 2A. Mean Observed and Estimated Gaps in CLA Scores between First- Generation and Non-First 
Generation Students 


Non-First Generation vs. First-Generation Gap in CLA Performance 
Observed Estimated 


Controlling for: 


Pre-college academic 

Pre-college academic and background 

First-year students 

Performance task 

-45.44** 

-2.37 

-1.91 

Argument 

-40.57** 

-11.13 

-9.87 

Make an argument 

-47.02** 

-15.18 

-14.16 

Break an argument 

-36.79** 

-9.23 

-6.72 

Seniors 

Performance task 

-26.86* 

0.96 

0.73 

Argument 

-23.27* 

6.58 

5.00 

Make an argument 

-30.6** 

-0.30 

-0.91 

Break an argument 

-12.69 

16.25* 

13.73 


**p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 

Note: Estimates are based on a regression model predicting different components of the CLA and controlling for students’ background characteristics 
and pre-college academic performance, as indicated. 
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Table 3A. Mean Observed and Estimated Gaps in CLA Scores between First-Generation and Non-First- 
Generation Students, in Urban and Non-Urban Settings 

Non-First Generation vs. First-Generation Gap in CLA Performance 
Observed Estimated 


Controlling for: 


Pre-college academic 

Pre-college academic and background 

NON-URBAN 




First-year students 




Performance task 

-52.34** 

-7.08 

-6.36 

Argument 

-52.55** 

-22.44 

-19.77 

Make an argument 

-64.11** 

-33.14** 

-32.07** 

Break an argument 

-40.90** 

-11.14 

-7.04 

Seniors 




Performance task 

-22.36 

-0.39 

0.79 

Argument 

-27.43 

1.33 

-1.92 

Make an argument 

-39.51* 

-11.20 

-12.11 

Break an argument 

-15.67 

13.12 

7.40 

URBAN 




First-year students 




Performance task 

-33.53 

2.26 

4.25 

Argument 

-21.26 

4.33 

4.05 

Make an argument 

-20.33 

9.53 

9.05 

Break an argument 

-28.52 

-6.74 

-5.27 

Seniors 




Performance task 

-27.37 

2.94 

-1.22 

Argument 

-15.76 

12.60 

13.92 

Make an argument 

-16.97 

12.10 

11.93 

Break an argument 

-6.94 

19.86 

22.27 


**p<0.01, *p<0.05 (significance tests adjusted for clustering of students within institutions) 

Note: Estimates are based on a regression model predicting different components of the CLA and controlling for students' background characteristics and pre- 
college academic performance, as indicated. 
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Table 4A. CLA Scores, by Pell-Grant Eligibility 



N 

Pell 

Mean 

SD 

N 

Non-Pell 

Mean 

SD 

Mean difference 
Raw scores 

Mean difference 
Effect size (SD) 

First-year students 

Performance task 

728 

1,025.42 

165.75 

1021 

1,054.62 

158.93 

-29.20** 

-0.18 

Argument 

678 

1,022.68 

136.65 

1014 

1,057.05 

135.63 

-34.37** 

-0.25 

Make an argument 

692 

1,016.04 

162.89 

1024 

1,054.07 

151.40 

-38.03** 

-0.24 

Break an argument 

683 

1,026.26 

154.87 

1020 

1,058.59 

158.65 

-32.33** 

-0.20 

Seniors 

Performance task 

445 

1,166.13 

164.23 

807 

1,179.22 

176.22 

-13.09 

-0.08 

Argument 

420 

1,146.99 

140.48 

750 

1,165.63 

142.89 

-18.64 

-0.13 

Make an argument 

421 

1,123.08 

157.05 

757 

1,145.52 

168.61 

-22.44 

-0.14 

Break an argument 

422 

1,170.45 

163.98 

753 

1,183.74 

162.14 

-13.29 

-0.08 


Note: **p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 


Table 5A. Mean Observed and Estimated Gaps in CLA Scores between Pell-Eligible and Non-Pell-Eligible 
Students 


Pell-Eligible vs. Non-Pell-Eligible Gaps in CLA Performance 
Observed Estimated 


Controlling for: 


Pre-coilege academic 

Pre-college academic and background 

First-year students 

Performance task 

-29.20** 

3.77 

6.37 

Argument 

-34.37** 

-8.53 

-2.31 

Make an argument 

-38.03** 

-10.33 

-3.47 

Break an argument 

-32.33** 

-8.29 

-1.15 

Seniors 

Performance task 

-13.09 

4.54 

8.22 

Argument 

-18.64 

2.23 

-0.96 

Make an argument 

-22.44 

-1.62 

-1.22 

Break an argument 

-13.29 

6.92 

-0.41 


**p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 

Note: Estimates are based on a regression model predicting different components of the CLA and controlling for students' background 
characteristics and pre-college academic performance, as indicated. 
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Table 6A. Mean Observed and Estimated Gaps in CLA Scores between Pell-Eligible and Non-Pell-Eligible 
Students in Urban and Non-Urban Settings 

Pell-Eligible vs. Non-Pell-Eligible Gaps in CLA Performance 
Observed Estimated 

Controlling for: Pre-college academic Pre-college academic and background 

NON-URBAN 

First-year students 

Performance task 

-20.84 

0.08 


3.93 

Argument 

-36.36** 

-16.44* 


-6.07 

Make an argument 

-40.37** 

-19.73* 


-7.61 

Break an argument 

-32.45* 

-13.32 


-5.36 

Seniors 

Performance task 

-28.69 

-7.72 


-5.19 

Argument 

-22.55 

4.37 


8.22 

Make an argument 

-33.29 

-6.64 


1.44 

Break an argument 

-11.33 

15.92 


16.01 

URBAN 

First-year students 

Performance task 

-37.28* 

4.85 


8.79 

Argument 

-28.61 

-0.20 


2.98 

Make an argument 

-31.26* 

1.10 


4.19 

Break an argument 

-29.04 

-4.54 


3.39 

Seniors 

Performance task 

17.27 

23.41 


28.03 

Argument 

-9.58 

0.18 


-11.25 

Make an argument 

-5.29 

4.23 


-4.03 

Break an argument 

-11.43 

-2.19 


-18.79 


**p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 

Note: Estimates are based on a regression model predicting different components of the CLA and controlling for students' background characteristics 
and pre-coilege academic performance, as indicated. 
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Table 7A. CLA Scores, by Race/Ethnicity 



African-American/Hispanic 

N Mean SD 

N 

White 

Mean 

SD 

Mean difference 
Raw scores 

Mean difference 
Effect size (SD) 

First-year students 

Performance task 

319 

973.24 

145.40 

1,293 

1,061.07 

162.06 

-87.83** 

-0.54 

Argument 

277 

991.78 

123.46 

1,296 

1,056.28 

137.67 

-64.50** 

-0.47 

Make an argument 

288 

984.57 

151.42 

1,306 

1,052.03 

156.59 

-67.46** 

-0.43 

Break an argument 

281 

991.91 

146.36 

1,302 

1,059.77 

158.25 

-67.86** 

-0.43 

Seniors 

Performance task 

141 

1,083.46 

161.81 

1,015 

1,195.60 

166.57 

-112.14** 

-0.65 

Argument 

118 

1,120.90 

148.26 

947 

1,168.56 

139.19 

-47.66* 

-0.34 

Make an argument 

119 

1,092.35 

165.60 

953 

1,148.09 

162.67 

-55.74* 

-0.34 

Break an argument 

119 

1,148.18 

169.21 

950 

1,187.47 

160.73 

-39.29 

-0.24 


Note: **p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 


Table 8A. Mean Observed and Estimated Gaps in CLA Scores between African-American/Hispanic and White 
Students 


African-American/Hispanic vs. White Gap in CLA Performance 


Controlling for: 

Observed 

Pre-college academic 

Estimated 

Pre-college academic and background 

First-year students 

Performance task 

-87.83** 

-9.25 


-4.56 

Argument 

-64.50** 

-13.92 


-13.89 

Make an argument 

-67.46** 

-11.78 


-15.54 

Break an argument 

-67.86** 

-21.16 


-16.45 

Seniors 

Performance task 

-112.14** 

-53.71* 


-46.05 

Argument 

-47.66* 

-15.24 


-14.86 

Make an argument 

-55.74* 

-23.07 


-25.96 

Break an argument 

-39.29 

-8.89 


-5.57 


**p<0.01, *p<0.0 5 (significance tests adjusted for clustering of students within institutions) 

Note: Estimates are based on a regression model predicting different components of the CLA and controlling for students’ background 
characteristics and pre-college academic performance, as indicated. 
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Table 9A. CLA Scores, by Urbanicity 



N 

Urban 

Mean 

SD 

N 

Non-Urban 

Mean 

SD 

Mean difference 
Raw scores 

Mean difference 
Effect size (SD) 

First-year students 

Performance task 

755 

1,033.15 

159.75 

994 

1,049.54 

164.11 

-16.39 

-0.10 

Argument 

723 

1,032.94 

133.00 

969 

1,050.99 

139.55 

-18.05 

-0.13 

Make an argument 

738 

1,026.41 

153.35 

978 

1,048.04 

159.49 

-21.63 

-0.14 

Break an argument 

730 

1,035.92 

156.75 

973 

1,052.90 

158.45 

-16.98 

-0.11 

Seniors 

Performance task 

581 

1,135.22 

177.38 

671 

1,208.64 

160.12 

-73.42* 

-0.43 

Argument 

540 

1,140.75 

148.47 

630 

1,174.53 

134.88 

-33.78 

-0.24 

Make an argument 

548 

1,115.28 

174.35 

630 

1,156.82 

153.68 

-41.54 

-0.25 

Break an argument 

543 

1,163.87 

166.95 

632 

1,191.94 

158.24 

-28.07 

-0.17 


Note: **p<0.01, *p<0.05 (significance tests adjusted for clustering of students within institutions) 


Table 10A. Mean Observed and Estimated Gaps in CLA Scores between Students in Urban and Non-Urban 
Settings 


Urban vs. Non-Urban Gap in CLA Performance 
Observed Estimated 


Controlling for: 


Pre-college academic 

Pre-college academic and background 

First-year students 

Performance task 

-16.39 

11.66 

6.66 

Argument 

-18.05 

-0.76 

-3.08 

Make an argument 

-21.63 

-3.13 

-4.81 

Break an argument 

-16.98 

-0.86 

-3.42 

Seniors 

Performance task 

-73.42* 

-52.18* 

-53.02* 

Argument 

-33.78 

-9.53 

-11.64 

Make an argument 

-41.54 

-17.44 

-19.53 

Break an argument 

-28.07 

-4.90 

-6.78 


**p<0.01, *p<0.05 (significance tests adjusted for clustering of students within institutions) 

Note: Estimates are based on a regression model predicting different components of the CLA and controlling for students' background characteristics 
and pre-college academic performance, as indicated. 
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Table 11A. Number of Cases and Performance Task Scores for First-Year Students across Three Samples 


CIC Full Sample 

CIC Representative Sample 

CAE Value-Added sample 

N Mean 

N Mean 

N Mean 


URBAN 


College 1 

152 

991 

152 

991 

142 

996 

College 2 

86 

991 

86 

991 

86 

991 

College 3 

89 

1,091 

89 

1,091 

89 

1,091 

College 4 

96 

991 

45 

1,012 

45 

1,012 

College 5 

79 

1,162 

57 

1,159 

57 

1,159 

College 6 

53 

1,061 

53 

1,061 

53 

1,061 

College 7 

98 

1,019 

44 

1,016 

44 

1,016 

College 8 

86 

1,015 

50 

1,030 

50 

1,030 

College 9 

17 

1,048 

17 

1,048 

14 

1,077 

NON-URBAN 







College 10 

95 

1,041 

68 

1,022 

68 

1,022 

College 11 

101 

985 

44 

997 

42 

1,006 

College 12 

94 

968 

94 

968 

94 

968 

College 13 

59 

989 

59 

989 

50 

982 

College 14 

145 

1,065 

145 

1,065 

132 

1,069 

College 15 

101 

1,115 

49 

1,109 

48 

1,112 

College 16 

93 

1,150 

42 

1,138 

38 

1,150 

College 17 

91 

1,039 

52 

1,044 

52 

1,044 

College 18 

108 

1,082 

53 

1,071 

51 

1,975 

College 19 

107 

1,029 

107 

1,029 

101 

1,039 
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Table 12A. Number of Cases and Performance Task Scores for Seniors across Three Samples 


CIC Full Sample 

CIC Representative Sample 

CAE Value-Added Sample 

N Mean 

N Mean 

N Mean 


URBAN 


College 1 

112 

1,075 

112 

1,075 

106 

1,078 

College 2 

54 

1,102 

54 

1,102 

52 

1,098 

College 3 

49 

1,203 

49 

1,203 

49 

1,203 

College 4 

64 

1,120 

48 

1,133 

48 

1,133 

College 5 

82 

1,218 

43 

1,184 

42 

1,186 

College 6 

41 

1,171 

41 

1,171 

22 

1,202 

College 7 

106 

1,126 

106 

1,126 

105 

1,126 

College 8 

63 

1,109 

41 

1,117 

28 

1,122 

College 9 

10 

1,184 

10 

1,184 

9 

1,207 

NON-URBAN 







College 10 

53 

1,082 

53 

1,082 

50 

1,072 

College 11 

61 

1,181 

42 

1,172 

42 

1,172 

College 12 

91 

1,142 

91 

1,142 

91 

1,142 

College 13 

40 

1,171 

40 

1,171 

35 

1,181 

College 14 

55 

1,167 

55 

1,167 

51 

1,167 

College 15 

107 

1,275 

54 

1,268 

54 

1,268 

College 16 

57 

1,305 

51 

1,298 

50 

1,302 

College 17 

75 

1,189 

41 

1,189 

44 

1,179 

College 18 

96 

1,274 

51 

1,286 

51 

1,286 

College 19 

36 

1,232 

36 

1,232 

36 

1,232 
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Table 13A. Institutional CLA Performance Levels 



CLA Total 

Performance Task 

Argument 

URBAN 

College 1 

Near 

Near 

Near 

College 2 

Near 

Near 

Near 

College 3 

Above 

Near 

Above 

College 4 

Near 

Near 

Near 

College 5 

Near 

Below 

Near 

College 6 

Near 

Near 

Near 

College 7 

Near 

Near 

Near 

College 8 

Near 

Near 

Near 

College 9 

Near 

Near 

Above 

NON-URBAN 

College 10 

Below 

Near 

Below 

College 11 

Near 

Near 

Near 

College 12 

Above 

Above 

Above 

College 13 

Above 

Above 

Near 

College 14 

Below 

Below 

Well Below 

College 15 

Near 

Near 

Near 

College 16 

Near 

Above 

Near 

College 17 

Near 

Near 

Near 

College 18 

Above 

Above 

Near 

College 19 

Near 

Near 

Near 


Source: Council for Aid to Education 
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